Printer Friendly

Using hierarchical linear modelling to examine factors predicting English language students' reading achievement.

In Canada, the ever-changing population landscape has a direct impact on the language spoken by students in elementary and secondary schools, while the language of education has remained English and French. As diverse immigrant groups have varying first languages, children of immigrants whose first language is not English often start school in English (2) as either ESL, EAL, or ELL learners. (3) To provide assistance to these student newcomers, school boards and provincial ministries of education provide a supportive environment through designated ELL classes in schools in addition to regular classes. Approaches for this language support vary from province to province, and the gains for students' English language proficiency development vary as well due to various factors influencing the language progress.

Learning "English" and "in English" for children whose first language is not English has been widely researched. Collier and Thomas (1989) described that during the acquisition process of another language, proficiency (i.e., English in this context) generally takes a significant number of years for newly immigrant students. Students of different language backgrounds who are newcomers in Canada may not perform as well as students who are native English speakers when it comes to national assessments that require certain reading skills. Mitchell, Destino, and Karam (1997) estimated that gaining proficiency in a newly learned language might take up to 10 years to attain. At the beginning of the learning process, the ELL student group is at a disadvantage: Woo (2009) found that non-English Language Learners (non-ELLs) tend to score higher than ELLs (i.e., first-generation immigrants who are just learning English) in the reading assessment of National Assessment of Educational Progress (NAEP), even when differences between student, teacher, and school backgrounds are controlled. (4) However, as time goes on, these ELL students whose first language is not English or who speak a variety of English significantly different than the type of English used for instruction in school (Ontario Ministry of Education, 2008) may "pick up" the language and eventually perform as well as their native English-speaking classmates. Jang, Dunlop, Wagner, Kim, and Gu (2013) found that after a certain time ELL students in fact outperformed the English-speaking students.

Different factors may contribute to a student's mastery of language in different ways. For example, the age when one immigrates seems to be a major factor in contributing to students' reading achievement. Recent large-scale assessment conducted by the Organisation for Co-operation and Development (OECD) recently demonstrated that 15-year-olds who have just arrived to a new country showed poorer reading achievement than students who immigrated at age 5 or younger (OECD, 2013), indicating the length of residency and exposure to the language could be a key factor in students' success on their reading development.

Although ELLs' language learning in school settings may itself be a major factor in contributing to students' language development, other factors should also be examined to get a full understanding of such development (Abedi 2004, 2008; Abedi & Levine, 2013). For example, the amount of English exposure at home and at school could also be contributing factors, as vocabulary knowledge is associated with reading achievement (Qian, 2002; Stahl, Chou Hare, Sinatra, & Gregory, 1991). Family factors such as socioeconomic status (SES), parental language preference, and personal literacy supports are also found to be related to reading achievement (D'Angiulli, Siegel, & Maggi, 2004; Duursma et al., 2007). The level of education of the parents, especially the mother, is also found to be a factor contributing the reading achievement of students (Rabbany, Zaiane, & ElAtia, 2014).

Many studies have been done in Canada with regard to students' reading achievement in various provinces. For instance, in the seminal work of Cummins (1981) focused on immigrants in Canada, he argued that language proficiency is not a unitary construct; specifically, some aspects of language proficiency, such as reading skills, are strongly related to cognitive and academic development (p. 132). Although Abedi's (2004) work is mostly with Hispanic ELL students in the United States, he warns against making generalizations about the ELL group, especially the LEP (limited English proficiency) subgroup. Variations within the ELL group, alongside the inconsistent classification of the group and the "dispersed" group, should be taken into consideration when interpreting the test results for making fair decisions.

The study that we undertook for this article separates itself from others by looking at a large set of data for a standardized low-stakes reading achievement test from across Canada, including all provinces regardless of the language situation. This test evaluated content and comprehension, and was not a language test per se. Our study aimed to shed some light on factors important to educators in understanding students' reading achievement within the multicultural context of Canada; it did not focus on one particular group. It further aimed to identify factors that predict students' reading performance on a national assessment, while taking into consideration non-native ELL and native English speakers.

Purpose of the Study

In this study, our main objective was to identify, by using Hierarchical Linear Modelling (HLM), factors such as ELL status that would predict students' reading performance in an English language arts exam from across Canada. We aimed to answer the following questions:

1. What factors within students' linguistic history, language exposure, and practice of language contribute to predicting their reading performance?

2. Which of these factors contribute most significantly to predicting reading performance?

3. What are the effects of school variables in predicting reading performance?

Results of this study can potentially provide new insights into factors influencing reading performance and competency for Grade 8 students. It may provide stakeholders (parents, educators, etc.) with a better understanding and use of assessment in order to address specific issues that relate to creating assistance programs for both students who are ELL and native L1 English speakers who are at risk of low reading performance. Results in the Canadian context may also help educators understand how to improve teaching and learning support for students from diverse language backgrounds.

Context of the Study

This study was conducted in the Canadian context. Data from a Canadian national assessment, the Pan-Canadian Assessment Program (PCAP), were used. The PCAP is a Canada-wide achievement test that assesses performance of 13-year-old (Grade 8) students on their reading, mathematics, and science skills. The PCAP was developed by the Council of Ministers of Education, Canada (CMEC) in attempt to provide large-scale assessment and reporting on students' achievement across Canada (CMEC, n.d.). The previous form of PCAP was the School Achievement Indicator Program (SAIP), which was administered nine times between 1993 and 2004. In 2003, a decision was made by CMEC to replace the SAIP with the PCAP due to changes in curriculums and the need to reflect students' learning across Canada. The test was administered using random sampling in both English and French to schools from across Canada.

The PCAP has two main purposes: "(a) to help ministries and departments of education make informed decisions about education, and (b) to determine how prepared 13-year-old students are for the academic challenges of high school" (CMEC, n.d., p. 1). As the entire purpose of the PCAP is to ensure similarity of student learning across Canada, in regards to preparation for high school entry, CMEC (n.d.) stressed that results of the test should not be used to reflect students' individual performance, nor should results be used to compare schools and jurisdictions. To avoid misuse of the test results, the PCAP results are not available to school boards, ministries of educations, schools, or teachers. The PCAP is administered every three years, and each administration focuses on one major core subject and two minor subjects, alternating between mathematics, reading, and science. Along with taking the assessment, students are required to respond to a questionnaire targeting detailed information such as reading habits and attitude. Additionally, a teacher questionnaire and a school questionnaire are also administered.

For this study, data from the 2007 administration of the PCAP was used. The core subject that year was reading, with mathematics and science as the minor subjects. Only the data from the English administration of the reading assessment were used, because the interest of this study was students' English reading achievement. The data were obtained from all 10 provinces and 1 territory (Yukon) in Canada. For the English reading assessment, 1,281 schools and 15,000 students aged 13 participated. Booklet 1 of the two reading booklets was used for this study; after removing missing cases, a total of 7,467 students remained in the data for analyses. (5) The results and the answers to the questionnaire provide a unique opportunity to study students' achievement by looking at various elements that could contribute to progress.

Assessing Reading: Between Language and Cognition

Assessing reading is not an easy task. It is complex and, according to Bachman (2000), is "the most extensively researched and most enigmatic of the so-called language skills" (p. x). During a reading task, two elements are happening: a process of reading and a product of reading--from deciphering the written codes and the meanings of words and sentences to understanding messages and connotations. Both elements are complex and involve multiple tasks and subtasks. Regardless of whether it is in a first language or a second/ foreign language, reading is a linguistic, cognitive, and metacognitive task that requires particular skills to happen.

While reading is challenging for English-speaking students, the challenge is even higher for the non-native ELL group. It is important to understand the complexities of this group. The subgroups that form this larger group are quite different and have different results with substantial repercussions on the development of reading skill.

1. The immigrant group is made up of families who chose to immigrate to Canada. While the main applicant must possess a higher level of education and a higher level of English competence in order to successfully immigrate, this is not generally the case for the accompanying spouse and/or children. Thus the children of these immigrant families may receive very few supports at home, posing a challenge for them to perform well in school. Jung et al. (2013) found that the amount of exposure to the target culture is a critical factor to non-native ELLs. Rabbany et al. (2014) found that the mother's level of education plays an important role in achievement on the reading portion of PCAP, for both the non-native ELL students and regular students whose first language is English.

2. The refugee group is entirely different from the immigrant group. In addition to unpredictable levels of education and linguistic proficiency of their parents, students may have a gap in their prior formal education as well as psychologically traumatic experiences. All these factors can severely hinder their learning progress and negatively impact their English language learning process.

The PCAP reading assessment provides a good overall view of students' performance as it provides not only the results of the tests, but also the surveys from schools, teachers, and students. Hence it gives an unique opportunity to analyze the intricate skill that is reading. For Alderson (2000) there are two levels of understanding: one is the literal understanding of the text presented to the reader, and the other is the "understanding of meanings that are not directly stated in text ... understanding of the main implications of text" (p. 7). Gray (1960) made three distinctions: one linked to the literal meaning (reading the lines), inferring meaning from text (reading between the lines), and higher critical thinking meaning (reading beyond the lines). In light of these distinctions alone, the results of the PCAP would be directly influenced by the students' linguistic and educational background. We need to question if ELL students have matured cognitively enough to be in the same reading age as the other examinees. Do they possess what Grabe (1991) refers to as the speed and automaticity of recalling words and linked words to meaning? Grabe lists the following six elements necessary for a fluent reader: (a) automatic recognition skills, (b) vocabulary and structural knowledge, (c) formal discourse structure knowledge, (d) content/world background knowledge, (e) synthesis and evaluation skills and strategies, and (f) metacognitive knowledge and skills monitoring.

From a metacognitive perspective, previous knowledge of the text or familiarity with the topic of the reading text (Fisher, Frey, & Lapp, 2012) may play a major role in test performance. The fact that the two groups we identified as ELL students (the immigrant group and the refugee group) may have either linguistic difficulties or educational gaps would play a major role in the development of the metacognitive skills necessary to understating texts. Abedi (2008) strongly advocates paying close attention to language proficiency when assessing for content among school students. In addition, items in the PCAP are culturally "Canadian." For instance, some items in the reading passages were about a game of hockey among school children. (6) Studies in differential item functioning found that previous knowledge of subjects in reading tests does influence test outcomes/performance (Recht & Leslie, 1988). The students in our two non-native ELL groups may not be familiar with the "cultural" texts of the reading assessment. We wonder to what extent such ELL factors impact their performance in the PCAP.

Methodology

Hierarchical Linear Modelling

Hierarchy Linear Modelling was chosen as the statistical procedure in this study due to the nested nature of the data. HLM is related to regression analyses as it provides a predictive relationship between predictor variables and the dependent variable. One benefit of HLM over the standard regression analyses is that it takes into consideration the multilevel (nested) structure of data. When data are nested, one should not ignore that and proceed with regression analyses. In linear and multiple regressions, observations should be considered as independent of each other; however, such cases could not be true at all times for nested data. Snijders and Bosker (2012) explained that any multilevel sampling design will cause dependent observations, "and failing to deal with this properly in the statistical analysis may lead to erroneous inferences" (p. 7). For example, students from the same classroom might be more similar to each other in terms of certain characteristics in comparison with other students from other classes, and such similarities should not be ignored when performing statistical analyses due to the possible effects of higher-level variables on the lower level. Other predictive variables and their associated errors at the higher levels might also affect the entire prediction model; therefore, it is wiser to use the HLM when multilevel data are involved. In this study, a two-level HLM model was proposed in which the students were nested within schools. Level 1 of the model was the student's individual level, in which variables directly associated with the students were examined for their relationship with reading achievement. Level 2 was the school level, which was chosen because schools as units were randomly chosen to take the PCAP, and students in this case are nested within the schools. After the removal of missing cases from the Level 2 data file, a total of 1,161 schools remained in the data for analyses. Table 1 illustrates the distribution of the data for schools across jurisdictions in Canada. The list of variables used in this study can be found in the Appendix. Examining the predictor variables at each of these two levels will provide a better idea of students' language background, foundations, and their relationships to the reading performance of students. The computer program HLM (Raudenbush, Bryk, & Congdon, 2004) was used in the analyses for this research. The dependent variable for the analyses was students' reading scores on the 2007 PCAP.

Selection of Predictors for the Models

The items on the student and school questionnaires were used to select predictors to enter into the HLM model. Due to the large number of items in the student questionnaire, the items were chosen based on support from the literature that such related predictors would contribute to students' achievement in reading. Before entering the predictors into the model, we first created an empty model to obtain the reliability estimate of the model, and to determine if the sample size was appropriate for the procedure. At Level 1, the predictors were based on the student level; school-related variables were at Level 2. The student questionnaire consisted of six sections, with a total of 39 items (plus subquestions of A, B, C, and D). At the student level (Level 1), several items that best represent the predictors were chosen (see Table 2).

Questions 1 to 12 in Table 2 each represent one student individual-level variable. For example, Question 6 represents student's individual level of participation in class. No composite variable was used in this study.

At the second level (the school level), three predictors were chosen: (a) the percentage of ELL students in the school, (b) the size/type of community the school was located in, and (c) the number of minutes of instructions per week in English Language Arts that the school offered 13-year-olds. These variables were chosen based on our interest in ELL students, on whether the amount of English Language exposure these students receive in school matters, and on whether the community size of the school location matters. The associated questions on the school questionnaire are shown in Table 3.

These predictors gave an idea of how language exposure and experiences at the school level may predict students' reading performance. The size of the community where the school is located was chosen as a predictor based on a study by Gershenson and Langbein (2015), which indicated that school sizes have an impact on students' mathematics and reading achievement. The larger the school's size the more negative impact it will have on students. Using the logic that larger schools tend to be located in larger communities, the community size predictor was chosen in this study. It was also believed that a smaller community might lead to a similarity among students' environmental background (i.e., rural or small town, and medium-sized town), and perhaps either a nurturing or a hindering language learning environment. Slavin and Madden (1999) suggested that students might have limited English proficiency not only because of difficulties in learning English per se, but also because these students might be new immigrants living in poverty. They add that those who were similar in terms of language, immigration status, and socioeconomic status may tend to stay within the same community. Therefore it would be worthwhile to examine community size as a variable. It was assumed that the smaller the community, the more similar the residents are in comparison to larger communities, and that students' reading achievement might be higher.

All of these predictors were then entered in HLM (version 6.06) for analyses. Categorical responses were first dummy coded before being entered into HLM. Grand mean centring was used when entering variables into the models. Results of using HLM would help to identify significant predictors on students' reading achievement (for more on the chosen variables, see Appendix).

Results

The Empty Model

The empty model was first performed, with the following equations:

Level-1 Model: [Y.sub.ij] = [[beta].sub.0j] + [R.sub.j]

Level-2 Model: [[beta].sub.0j] = [[gamma].sub.00] + [[upsilon].sub.0]

Combined Model: [Y.sub.ij] = [[gamma].sub.00] + [[upsilon].sub.0] + [R.sub.j]

Reliability estimate obtained from the empty model was 0.400, indicating a moderate low number. This could possibly be due to the lower number of students in each school after the deletion of missing cases.

As a next step, Tables 4 and 5 were compared to determine if the sample size of the data was adequate.

The numbers in Tables 4 and 5 are almost exactly the same, indicating that the theoretical estimates were similar to the estimates obtained from the data. These data provided a good sample size for the model. Intraclass correlation calculated using sigma squared ([[sigma].sup.2]) and Tau ([tau]) obtained

Sigma_squared = 7187.4136

Tau

INTRCPT1,B0 = 1268.5593

[rho]^ = [[tau].sup.2]/([[tau].sup.2] + [[sigma].sup.2])

The intraclass correlation demonstrated that 15.00% of the variance was over the student level, suggesting a multilevel nature of the data and providing a justification in testing additional levels of predictors. Table 6 demonstrated that U0 was significant, meaning that the intercepts of the schools were significantly different from each other ([x.sup.2] = 1965.43, p <.01).

Model with Added Predictors

In this step, all the Level 1 and Level 2 predictors were entered into the model. Level-1 Model

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

Level-2 Model

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

At Level 2, the percentage of ELL students in the school, the size/type of community the school was located in, and the minutes of instruction per week were included as predictors that would affect [[beta].sub.0], the group mean. The percentage of ELLs and the type of community of the school were included as predictors that were believed to affect the slope of ESL_FSL ([[beta].sub.2]; whether students have taken an ELL class). The same Level 2 variables were also assumed to be affecting the slope of LANGSCHE ([[beta].sub.10]), or "whether English is the language most school subjects were taught in." The variable on the type of community the school is located in was assumed to be affecting the slope of LANGHOME ([[beta].sub.9]), or "whether English is the language spoken at home." In addition, the minutes of instruction were believed to be affecting the slopes of ELA_HW ([[beta].sub.3]; the amount of time doing English Language Art [ELA] homework), and ClASSDIS ([[beta].sub.5]; participate in class discussion). Table 7 displays the results of the HLM analyses.

To summarize the results, [[beta].sub.0] was significant, meaning the group means differed significantly. The three Level 2 variables (percentage of ELL students in school, the type/size of community the school was located in, and the minutes of instruction per week on ELA class) did not contribute significantly in making such difference (p > .05). For the Level 1 predictors, 7 out of 10 predictors were significant predictors of students' reading scores (p < .05). These significant predictors are marked in bold in Table 7. Of all the significant predictors, the greatest t-ratio of READPAR suggested that parents' encouragement to read had the greatest effect on students' reading performance.

A positive coefficient for these predictors indicated that the higher the predictor's value, the higher the students' predicted reading scores. On predictors that were not treated as continuous variables (i.e., LIBRARY, LANGHOME, LANGSCHE), because the answer "No" was coded as "0" and "Yes" was coded as "1," a positive coefficient indicated that the performance score increased by the number of points indicated on the coefficient, on average, for those who answered the question as "Yes", when other variables are controlled for in the model (as in the case for LANGHOME and LIBRARY). On the other hand, when the coefficient is negative, those who answered a "Yes" answer lowered the expected performance score by the points indicated on the coefficient, on average, when other variables are controlled for in the model(as in the case of LANGSCHE). In terms of Level 2 variables that contributed significantly in affecting the slopes of Level 1 variables, only MIN_INST and COMM_TYP were significant for predictors ELA_HW and LANGSCHE respectively. However, ELA_HW was not a significant predictor of student scores.

The combined model with all the predictors is as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

The combined model with only significant predictors is as follows:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

Explained Variance

Total variability from empty model = [[tau].sup.2] + [[sigma].sup.2] = 1268.55926 + 7187.41360 = 8455.97

Variability after predictors were added: = 946.96966 + 6452.09659 = 7399.07

Percentage of explained variance = 7399.07/8455.97 = 0.88 = 1 - 0.88 = 0.12 [right arrow] 12%

The percentage of explained variance suggested that the Level 1 predicting variables contributed 12% to the model.

Discussion and Conclusion

This research utilized HLM to identify the predictive relationship between students' linguistic history, language exposure, and practice of language on reading performance. Using the 2007 PCAP data, results demonstrated that at the student level, the number of individual projects requiring students to work outside of class, student's participation in class discussion during class, English being the language spoken at home (which was positively correlated with reading achievement), how often the parents encourage the student to read, age a child first learned to read, whether students like to use the library as a place for study or reading, and whether English is the language most school subjects were taught in were found to be predictors of statistical significance. These predictors contributed 12% of the variability in the model, and how often parents encouraged students to read had the greatest effect of all significant predictors on reading achievement. It was surprising that ELL did not contribute significantly as a predictor of students' reading scores. However, it should be noted that this variable did not take into consideration how long the students had been enrolled into the ELL classes. The duration which students were taking ELL classes may contribute on different levels of reading skills.

At Level 2, all three predictors (i.e., percentage of ELL students in school, the type/size of community the school located in, and the minutes of instruction per week on ELA) were found to be insignificant in predicting students' reading scores. And since the intraclass correlation indicated that having a second (school) level in the analyses was necessary, the results of this study calls for selection of better Level 2 predictors in future research.

The results of this study provided useful information regarding the extent to which students' language exposure and experiences contributed to predicting their reading achievement. However, one should note that only about 12% of the variability was accounted for by these variables, indicating a possible underrepresentation of students' language profile. More significant predictors should be added in order to get a full picture, especially at Level 2, to better predict students' reading achievement. Future research may look at other factors outside of students' language background, for example, the socioeconomic status of families, mother's level of education, the province in which the school is located, or the size of classrooms. The result of low variability might also be due to the lack of representation of the predictors by the chosen items. Because only one survey item was selected to represent one category of predictor in this study, future research may group the items together based on categories to create composite variables, and obtain a category score representing the mean of the grouped items together. This step would ensure that the majority of the survey items are considered when entering data into the model, creating a better representation of the predictor. In regards to our interest in the ELL group, although this study has found that being in ELL classes did not contribute significantly to predicting students' reading performance, as discussed above, factors such as duration of students' enrolment in ELL class, and for non-Canadian-born students the age they moved to Canada and the amount of support received within and outside of their families, could be factors worth examining. In this study we have only examined the effects on reading performance; we do not know how the same variables may affect writing, which always goes hand in hand with reading in terms of language assessment. Future studies may also look at the writing outcomes of students.

The use of a Canadian performance measure (the PCAP) allowed results to be applied in a multicultural framework. It could further help various stakeholders in assisting students to improve their reading abilities, especially for ELL students who are newcomers to an English-speaking country. In addition, parents and teachers could benefit from these results and improve students' reading environments and habits. For example, teachers may create more targeted opportunities for students to practice their reading skills, hence allowing them more exposure for gaining reading experiences within the school. Parents and/or guardians of ELL students may also implement strategies at home for encouraging reading in English in order to help their children with English reading proficiency, and for better understanding of language learning processes within contents and subject areas. Although more predictors have yet to be identified, this study serves as a first step in understanding the factors on predicting students' reading achievement.

Future research could also investigate the age of the students taking the assessment as another important factor for consideration. Such studies would be consistent with the results and conclusions of Cummins (1981) in regards to whether or not the age of ELL students is a contributing factor to their reading performance. Students at this age, 13-years-old, are at a critical period for learning, from the perspective of both second language acquisition (SLA) theory and cognitive psychological development. In order to advance both fields in understanding how reading as a literacy skill intertwines with language acquisition, a longitudinal study spanning three years--from a year before taking the PCAP to two years after sitting for the PCAP--may contribute to greater understanding of the effect of age.

Acknowledgements

Dr. ElAtia would like to thank the Social Sciences and Humanities Research Council of Canada for the grant that allowed her to conduct this research. She would also like to thank Dr. Kathryn O'Grady, manager of the Pan-Canadian Assessment Program, and Mr. Pierre Brochu, Director of the Learning Assessment Programs Division of the Council of Ministers of Education, Canada, for their assistance and feedback, and for providing the necessary data to carry out this project.

Both authors would like to thank the two anonymous reviewers of earlier versions of this article for their valuable comments and suggestions.

The Authors

Karen Fung is a Ph.D. student specializing in Measurement, Evaluation, and Cognition in the Department of Educational Psychology at the University of Alberta. Her research interests include automatic item generation, automatic essay scoring, test translations and adaptations, and educational and psychological measurement. She is also a psychometrician currently residing in Toronto.

Dr. Samira ElAtia is Associate Professor of Language Assessment and Education and currently Director of Graduate Studies at Faculte Saint-Jean, University of Alberta. She is also a member of the Board of Directors for the Centre for Canadian Language Benchmarks. Her research interest focuses on sociolinguistics biases in assessing English and/or French as second/foreign language.

Karen Fung & Samira ElAtia (1)

Notes

(1) The corresponding author and project leader of this study.

(2) For the purpose of this article, French and francophone immigrants are not included in this study.

(3) School-age children in K-12 who are learning English in an ESL or EAL setting are commonly referred to as ELL--English Language Learner/Learning (Abedi & Gandara, 2007; Abedi, Hofstetter, & Lord, 2004). This distinguishes them from students learning English as a language art subject who speak English at home as their first/native language. English is not the language ELL students identify with outside school, and they have varying degrees of competence in it. In this article, and for saliency purposes, we will be using ELL for students who are learning English but are not native speakers of English; to them English could be a second language, or it could be an additional language to one or more languages already spoken at home.

(4) "ELL students" refers to students in K-12 schooling as opposed to ELL adults.

(5) The First Nations group was not being studied in this article due to its small sample size in the PCAP data.

(6) We cannot share real test items as we are bound by a confidentiality agreement with CMEC.

(7) Due to the amount of dummy coding required for certain categorical variables, which would result in a higher number of additional predictors, the authors have changed the forms of the questions into dichotomous responses.

(8) The term "ESL" here is copied directly from the PCAP documents; we opted to keep the questions as they appear in the documentation. It is used as such when we ran the analyses as well.

References

Abedi, J. (2004). The No Child Left Behind Act and English Language Learners: Assessment and accountability issues. Educational Research, 33(1), 4-14.

Abedi, J. (2008). Measuring students' level of English proficiency: Educational significance and assessment requirements. Educational Assessment, 13(2-3), 193-214.

Abedi, J., & Gandara, P. (2007). Performance of English language learners as a subgroup in large-scale assessment: Interaction of research and policy. Educational Measurement: Issues and Practice, 24(4), 36-46.

Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1-28.

Abedi, J., & Levine, H. (2013). Fairness in assessment of English Learners. Leadership, 42(3), 26-28.

Alderson, J. C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press.

Bachman, L. F. (2000). Preface. In J. C. Alderson, Assessing reading. Cambridge, UK: Cambridge University Press.

Collier, V. P., & Thomas, W. P. (1989). How quickly can immigrant become proficient in school English? Journal of Educational Issues of Language Minority Students, 4, 26-38.

Council of Ministers of Education, Canada [CMEC]. (n.d.). Pan-Canadian assessment program (PCAP). Retrieved from http://www.cmec.ca/240/Programs-and-Initiatives/Assessment/ Pan-Canadian-Assessment-Program-(PCAP)/Overview/index.html

Cummins, J. (1981). Age on arrival and immigrant second language learning in Canada: A reassessment. Applied Linguistics, 2(2), 132-149.

D'Angiulli, A., Siegel, L. S., & Maggi, S. (2004). Literacy instruction, SES, and word-reading achievement in English-language learners and children with English as a first language: A longitudinal study. Learning Disabilities Research & Practice, 19(4), 202-213.

Duursma, E., Romero-Contreras, S., Szuber, A., Proctor, P., Snow, C., August, D., & Calderon, M. (2007). The role of home literacy and language environment on bilinguals' English and Spanish vocabulary development. Applied Psycholinguistics, 28(1), 171-190.

Fisher, D., Frey, N., & Lapp, D. (2012). Building and activating students' background knowledge: It's what they already know that counts. Middle School Journal, 43(3), 22-31.

Gershenson, S., & Langbein, L. (2015). The effect of primary school size on academic achievement. Educational Evaluation and Policy Analysis, 37(1), 135S-165S.

Grabe, W. (1991). Current developments in second-language reading research. TESOL Quarterly, 25(3), 375-406.

Gray, W. S. (1960). The major aspects of reading. In H. Robinson (Ed.), Sequential development of reading abilities (pp. 8-24). Chicago, IL: Chicago University Press.

Jang, E., Dunlop, M., Wagner, M., Kim, Y., & Gu, Z. (2013). Elementary school ELLs' reading skill profiles using cognitive diagnosis modeling: Roles of length of residence and home language environment. Language Learning, 63(3), 400-436.

Mitchell, D., Destino, T., & Karam, R. (1997). Evaluation of English language development programs in the Santa Ana Unified School District: A report on data system reliability and statistical modeling of program impacts. Riverside, CA: California Educational Research Cooperative, University of California, Riverside. Retrieved from http://cerc.ucr.edu/publications

Ontario Ministry of Education. (2008). Supporting English language learners: A practical guide for Ontario educators-Grade 1 to 8. Toronto, ON: Queen's Printer for Ontario. Retrieved from http://www.edu.gov.on.ca/eng/document/esleldprograms/guide.pdf

Organisation for Economic Cooperation and Development. (2013, June). PISA in focus. Retrieved from http://www.oecd.org/pisa/pisaproducts/pisainfocus/pisa in focus n29 (eng)--Final.pdf

Qian, D. D. (2002). Investigating the relationship between vocabulary knowledge and academic reading performance: An assessment perspective. Language Learning, 52(3), 513-536.

Rabbany, R., Zaiane, O. R., & ElAtia, S. (2014, August). Mining large scale data from National Educational Achievement Test. Presentation at the KDD Workshop on Data Mining for Educational Assessment and Feedback, in conjunction with ACM SIGKDD 2014, New York City, NY.

Raudenbush, S. W., Bryk, A. S, & Congdon, R. (2004). HLM 6 for Windows [Computer software]. Skokie, IL: Scientific Software International.

Recht, D. R., & Leslie, L. (1988). Effect of prior knowledge on good and poor readers' memory of test. Journal of Educational Psychology, 80(1), 16-20.

Slavin, R. E., & Madden, N. A. (1999). Effects of bilingual and English as a second language adaptations of Success for All on the reading achievement of students acquiring English. Journal of Education for Students Placed at Risk, 4(4), 393-416.

Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Thousand Oaks, CA: Sage.

Stahl, S. A., Chou Hare, V., Sinatra, R., & Gregory, J. F. (1991). Defining the role of prior knowledge and vocabulary in reading comprehension: The retiring of number 41. Journal of Reading Behavior, 23, 487-508.

Woo, H.-J. (2009). Investigating on multiple factors affecting English-language learners' reading achievement: Hierarchical linear modeling approach (Doctoral dissertation). Retrieved from https:// www.ideals.illinois.edu/bitstream/handle/2142/14743/1_Woo_Hui-Jeong.pdf?sequence=3
Appendix. Representation of Variable Names

Question                                              Variable Name

Were you born in Canada?                              CAN_BORN

Have you ever been in an                              ESL_FSL
English-as-a-Second-Language program or course?

What language is most often spoken in your home?      LANGHOME
(Please check ONLY ONE RESPONSE)

On average, how much time do you spend in total       ELA_HW
EACH WEEK on homework in English Language Arts?

In your English Language Arts classes, how often      PROJ_OUT
do you have the following kinds of assignments?
Individual projects requiring work outside of
class

How much does each of these apply to reading in       CLASSDIS
your English Language Arts classes? I participate
in class discussions in English Language Arts

How often did your parents or guardians do these      READPAR
things when you were younger? They encouraged me
to read

When did you first learn to read? What language       FIRSTREA
are most subjects taught in school? (Please check
ONLY ONE RESPONSE)

How would you describe your school library? It is     LIBRARY
a place I like to go to read or to study

What percentage of students in your school are        PERCENT
identified as English as a Second Language (ESL)
learners?

In what type of community, town, or city is your      COMM_TYP
school located? (For this question, please think
of what is generally considered locally to be your
community, even if it is located near a larger
town or city.)

How many minutes of instruction per week does your    MIN_INST
school offer in an English Language Arts class for
13-year-olds?

Table 1
School Participation by Jurisdiction

                             Number of schools   Percentage
Jurisdiction                   participating     in sample

Alberta                             133             11.5
British Columbia                    129             11.1
Manitoba                            151             13.0
New Brunswick                       104             9.0
Newfoundland and Labrador           102             8.8
Nova Scotia                         110             9.5
Ontario                             130             11.2
Prince Edward Island                28              2.4
Quebec                              78              6.7
Saskatchewan                        189             16.3
Yukon                                7              0.6

Table 2
Selected Items for HLM Model at Level 1 (Student Level)

Question                                 Options

1.     Were you born in Canada?          Yes / No
2.     Have you ever been in an          Yes / No
       English-as-a-Second-Language
       program or course?
3.     What language is most often       English                  [ ]
       spoken in your home? (Please      French                   [ ]
       check ONLY ONE RESPONSE)          Canadian Aboriginal
                                           (e.g., Cree,
                                           Inuktitut)             [ ]
                                         Other (e.g., German,
                                           Mandarin)              [ ]
4.     On average, how much time do      Less than 30 minutes     [ ]
       you spend in total EACH           30 minutes to one hour   [ ]
       WEEK on homework in English       One to two hours         [ ]
       Language Arts?                    More than two hours      [ ]

5.     In your English Language Arts     Rarely or never          [ ]
       classes, how often do you         Sometimes                [ ]
       have the following kinds of       Often                    [ ]
       assignments? Individual
       projects requiring work
       outside of class

6.     How much does each of these       Not at all               [ ]
       apply to reading in your          A little                 [ ]
       English Language Arts classes?    More than a little       [ ]
       I participate in class            A lot                    [ ]
       discussions in English
       Language Arts

7.     How often did your parents or     Rarely or never          [ ]
       guardians do these things         Sometimes                [ ]
       when you were younger?            Often                    [ ]
       They encouraged me to read

8.     When did you first learn to       Before kindergarten      [ ]
       read?                             Kindergarten             [ ]
                                         Grade 1                  [ ]
                                         Grade 2                  [ ]
                                         Later than Grade 2       [ ]
                                         I don't remember         [ ]

9.     What language are most            English                  [ ]
       subjects taught in school?        French                   [ ]
       (Please check ONLY ONE            Canadian Aboriginal
       RESPONSE)                           (e.g., Cree,
                                           Inuktitut)             [ ]
                                         Other (e.g., German,
                                           Mandarin)              [ ]

10.    How would you describe your       Yes / No                 [ ]
       school library?
       It is a place I like to go to
       read or to study
11.    Is English the language spoken    Yes / No
       at home?7
12.    Is English the language most      Yes / No
       school subjects were taught in?

Table 3
Selected Items for HLM Model at
Level 2 (School Level)

Question                            Options

What percentage of students in      [ ] 0% to 25%
your school are identified as       [ ] 26% to 50%
English as a Second Language        [ ] 51% to 75%
(ESL (8)) learners?                 [ ] 76% to 100%]

In what type of community, town,    [ ] Rural community or small
or city is your school located?         town (fewer than 5,000
(For this question, please think        people)
of what is generally considered     [ ] Medium-sized town (5,000
locally to be your community,           to 25,000 people)
even if it is located near a        [ ] Small city (25,000 to
larger town or city.)                   100,000 people)
                                    [ ] Medium city (100,000 to
                                        500,000 people)
                                    [ ] Large city (over 500,000
                                        people)
How many minutes of instruction
per week does your school offer
in an English Language Arts
class for 13-year-olds?

Table 4
Final Estimation of Fixed Effects

                  Standard
Fixed effect     coefficient   Approx. error   t-ratio    df    P-value

For
INTRCPT1, B0
INTRCPT2, G00    507.576956      1.686465      300.971   1113    0.000

Table 5
Final Estimation of Fixed Effects (with Robust Standard Errors)

                  Standard     Approx.
Fixed effect     coefficient    error     t-ratio    df    P-value

For
INTRCPT1, B0
INTRCPT2, G00    507.576956    1.686262   301.007   1113    0.000

Table 6
Final Estimation of Variance Components

             Random
             effect    Standard                        Chi-      P-
            deviation  component   Variance    df     square    value

INTRCPT1,   [U.sub.0]  35.61684   1268.55926  1113  1965.43010  0.000
Level-1         R      84.77862   7187.41360

Table 7
Final Estimation of Fixed Effects (with Robust Standard Errors)

Standard fixed              Approx.                               P-
effect                    coefficient    Error    t-ratio   df   value

For INTRCPT1, B0
  INTRCPT2, G00           492.014167   11.278748  43.623   1110  0.000
  PERCENT, G01              9.134706   19.071780   0.479   1110  0.632
  COMM_TYP, G02            -0.931712   5.349020   -0.174   1110  0.862
  MIN_INST, G03            -0.013325   0.008409   -1.585   1110  0.113

For CAN_BORN slope, B1
  INTRCPT2, G10             8.538074   6.044707    1.412   4888  0.158

For ESL_FSL slope, B2
  INTRCPT2, G20             2.196955   2.402140   0.91 5   4888  0.361
  PERCENT, G21             -7.648031   6.268568   -1.220   4888  0.223
  COMM_TYP, G22            -0.236334   1.647158   -0.143   4888  0.886

For ELA_HW slope, B3
  INTRCPT2, G30            2.2881 98   1.378027    1.660   4888  0.096
  MIN_INST, G31            -0.018624   0.009170   -2.031   4888  0.042

For PROJ_OUT slope, B4
  INTRCPT2, G40            8.21 0754   1.873386    4.383   4888  0.000

For CLASSDIS slope, B5
  INTRCPT2, G50            14.462192   1.330920   10.866   4888  0.000
  MIN_INST, G51             0.007798   0.005876    1.327   4888  0.185

For LIBRARY slope, B6
  INTRCPT2, G60            -5.328740   2.482299   -2.147   4888  0.032

For READPAR slope, B7
  INTRCPT2, G70            27.215317   1.976371   13.770   4888  0.000

For FIRSTREA slope, B8
  INTRCPT2, G80            -9.906087   1.331515   -7.440   4888  0.000

For LANGHOME slope, B9
  INTRCPT2, G90            18.075113   5.760337    3.138   4888  0.002
  COMM_TYP, G91            -0.227859   3.248130   -0.070   4888  0.945

For LANGSCHE slope, B10
  INTRCPT2, G100          -12.876626   4.531055   -2.842   4888  0.005
  PERCENT, G101            -6.433678   15.600035  -0.412   4888  0.680
  COMM_TYP, G102            7.055811   3.532597    1.997   4888  0.045
COPYRIGHT 2015 TESL Canada
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2015 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Fung, Karen; ElAtia, Samira
Publication:TESL Canada Journal
Article Type:Report
Geographic Code:1CANA
Date:Jun 22, 2015
Words:7263
Previous Article:Building teachers' assessment capacity for supporting English language learners through the implementation of the STEP language assessment in Ontario...
Next Article:The relationship between lexical frequency profiling measures and rater judgements of spoken and written general English language proficiency on the...
Topics:

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters