Printer Friendly

Crowdsourcing feedback.


Crowdsourcing involves canvassing for work or funding, typically via the internet, from a crowd of people. The word combines 'crowd' and 'outsourcing'. Using the principle that 'more heads are better than one', crowdsourcing can engage a large crowd of people for creative ideas, skills, knowledge and/ or participation. Well known examples of crowdsourcing include: Wikipedia, the Arab Spring (Egypt's uprising in 2011); the toy company, Lego, that invites users to design new products and then other users vote (see Figure 1); Airbnb; Greenpeace canvassing for the 'Let's Go' ads in an environmental protest.

Crowdsourcing feedback in education involves feedback to learners from multiple perspectives, including peers and self. Many researchers have demonstrated the effects of regular and multiple forms of feedback by teachers and peers to improve learning (Black & Wiliam, 1998; Hattie, 2009; Hattie & Timperley, 2007; Wiliam, 2014). What if teachers could ensure feedback to their students by crowdsourcing it? What would it mean for student learning? What would it mean for teacher workloads? This paper will explore these questions using the example of Scholar, a web learning environment, developed by Bill Cope and Mary Kalantzis (2013) from the University of Illinois and the Common Ground Research Networks, Champaign, Illinois. Some comments from students and teachers about their experiences of crowdsourced feedback are included.

Classroom interactions

One form of feedback in many classrooms is through the traditional Initiate-Respond-Evaluate (I-R-E) approach (Cazden, 2001). This is where the teacher asks a question, the student responds and the teacher provides feedback on whether the response is correct or not by evaluating it. Similarly, the teacher providing formative and/or summative feedback on an assignment, test or project puts the responsibility on the teacher. Crowdsourcing feedback, on the other hand, challenges existing forms of feedback to create a collaborative environment where learners learn from each other as well as from the teacher. Figure 2 demonstrates how feedback works in a traditional I-R-E classroom and in a collaborative environment.

New technologies are creating more opportunities for interactions that are learner to learner as well as learner to teacher, and where there are many interactions simultaneously. These include collaborative environments such as online discussion forums and blogs where learners give feedback by affirming, elaborating, questioning and challenging other students' responses, traditionally the work of the teacher.

An example of crowdsourcing feedback

In the following example of crowdsourcing in an online discussion, 10 of 66 responses are included. They are written by Year 7 students as they imagine the worst place in the world in which to live. This is a frontloading activity for a study of the novel Trash by Andy Mulligan (2011). This online forum is based on a social media environment where students read the comments and then at the bottom, add their own comments. In this way, the comments of other students prompt further thinking as well as providing models of how to respond, for students who are less confident. Another advantage is how much text students read in order to comment on other students' responses.
   The worst place in the world is a place that war and homelessness
   is extremely high, this place would also have limited food that
   would only be provided to the higher rich people. This place is so
   awful because of all the war and lack of food, this place also has
   a high rate of homelessness. For one, not having a home, or even
   shelter would be awful. There would be no food or even internet
   connection, and you would be in fear of your life. For two, living
   in a place surrounded by war would be horrible. You would literally
   die so easily from so many reasons. (Nyah)

   I think living on the street with no food or water with nobody
   around you is the worst place to live in. You have no shelter, food
   or water and you would be poor. You would have nothing and you
   would have to find or fight for things to live by. Nobody would
   want to live with nothing or no one. (Charley)

   Somewhere with no positivity or happiness. It's always raining and
   there are no bright colours. Nobody has fun and everyone takes
   things seriously or in a wrong way. Somewhere with no freedom. I
   think others would feel the same as me because nobody likes being
   sad or in a negative space. (Georgia)

   On the street because there is no wifi and how can I charge my
   phone? (Kieralee)

   The worst place I can imagine to live in, is any areas made of ice.
   Any place with a high rate of homelessness, war, starvation, or dry
   places with no water, could kill a human easily with frost bite,
   starvation, de-hydration, infection, sickness/illness, etc. I
   wouldn't want to live here because it's a higher risk of death,
   barely any or no chance to work or earn money, lower education, no
   home, no food or water, and illnesses. (Becca)

   @Charley I really like your comment. It really makes sense and I
   don't want to live there. (Nyah)

   @ Nyah, I definitely agree with you, war would be a horrible place
   to live in and nobody would want to live there. People are sick,
   they die around you and that can kill you too. Do you think the
   internet connection is really that important? (Charley)

   @Charley, Yeah I think living in the street would be a bad place to
   live and I agree because it would be horrible to have nothing and
   no one. (Bree)

   Nowhere would be the worst place to live. It would be really awful
   to have nowhere to live because you would have no roof over your
   head barely any clothes on your back and in winter it would be
   freezing. I think others would feel the same way. (Sam)

   In a desert. The sun is very strong. There are no trees to hide
   from the sun. There is no water which makes your life more
   difficult to survive. It's also very hard to find food. Sometimes
   there are very heavy dust storms and it's very hard to hide from
   them. Also there are a lot of people that have died over there,
   from lots of diseases that have been caught. Also there is no
   electricity and it's very hard to keep up with today's technology.

This was followed by responses to YouTube clips about the lives of people living in poverty. Then, rather than memory work, the students undertook knowledge work, adding more video clips and website links to their learning community. Students were reading, problem-solving and thinking critically. Such discussions go beyond I-R-E, as everyone has a voice and everyone is included and expected to respond. As one student pointed out, 'I get to hear everyone else's opinions and based on theirs, you can create your own post.'

The teacher still has a role, monitoring the discussion: 'The kids like to be able to talk to each other, but they are thinking more than they would in a regular Facebook kind of setting; they realise that the teacher can see this as well. There's a certain helpful guardedness before they post, and thoughtfulness before they post it, which is a good skill for us to teach kids in the technology age.' Also, comments are public and trackable, enabling the teacher to target support to students who need it.

Giving students a voice not only engages them, but it democratises the classroom discourse, as anyone can initiate discussions and contribute knowledge to the community (Kline, Letofsky, & Woodward, 2013). Open-ended questions also promote thinking, analysis, evaluation and creativity, and through writing blog-like responses, students move from the grammar of speaking to the grammar of writing, turning thinking and oral discussion into more academic writing (Kalantzis, Cope, Chan, & Dalley-Trim, 2016, p. 418). Further, as writing becomes more public and purposeful, students develop new relationships with their peers and there is a shift to higher participation and collaboration in the classroom (Lammers, Magnifico, & Curwood, 2014).

Structured feedback

In order to crowdsource feedback, teachers need to support students to give effective feedback. Rubrics which are used extensively in classrooms provide structured feedback and scaffold understanding of the quality of a work for students. However, to be effective as a crowdsourcing tool, the rubric must be prospective--forward looking and formative, rather than retrospective, looking back or summative--so that the student can offer specific feedback for improvement. The following example on text cohesion is taken from a Scholar rubric for writing a persuasive text, based on the Australian Curriculum: English in Year 8 (Australian Curriculum, Assessment and Reporting Authority, 2016). The rubric's other criteria include content and structure, persuasive devices, language features, and use of multimedia. Note the scaffolding for students to offer specific feedback. The rating descriptions also support them to make a judgement and then justify the rating they assign--again with support.

Text cohesion:

Does the writer use a range of cohesive devices (connectives, topic sentences/openers, and concluding sentences/closers) to signpost ideas and make connections between ideas?

Are the paragraphs organised logically, each dealing with a single topic/idea? Are the sentences structured so that the beginning of the sentence focuses the reader's attention on the main message/ idea (theme position)?

Are synonyms, antonyms and repetition of words used to build associations between ideas?

Reviewers: Make suggestions for how to make the structure of the text more cohesive. Use the Annotations Tool to make change suggestions to make the text more cohesive at the word and sentence levels.

Rating descriptions

0 The use of cohesive devices is limited, making it difficult for the reader to follow the line of argument.

1 The writer has followed a clear structure, but needs to deal with a single topic/idea in each one. More connectives (e.g., in other words, consequently, finally, furthermore, for example, on the other hand) will also make the text more cohesive.

2 The writer has followed a clear structure in each paragraph, using openers, closers and connectives effectively, and each paragraph deals with a single topic/idea. Work at the sentence level (theme at beginning of sentence) and at the word level (antonyms, synonyms, repetition) will help to make this text more cohesive.

3 The structure, paragraphs, sentences, vocabulary, and use of connectives make this a very cohesive and coherent text so that the reader could easily follow the line of argument and be persuaded.

Such detailed rubrics can be made accessible to students with technology. In the example in Figure 3, a student has submitted the first draft of a research report on the effects of global warming on the survival of polar bears. This work is guided by a rubric that focuses on the research focus, sources, evidence, and interpreting and integrating information. The reviewer has then used the same rubric to offer feedback that might help the peer in the revision of their work. Being a reviewer also positions students as experts in their disciplines--as authors, artists, art critics, journalists, scientists, or as critical readers or writers.

In the next version (see Figure 4), the student has taken on the feedback from the reviewer and is ready to submit a revision. The student has also written a self-review, showing metacognition of what makes a good research report. This learning can also be applied to the next research report that the student will write. The writing space on the left of the screen represents cognition where the focus is on understanding facts and information; the right hand column with the rubric represents metacognition where the emphasis is on thinking about thinking and critical self-reflection.

Providing structured peer feedback exposes learners to different perspectives and ways of thinking. Also, in this project about the environmental effects of global warming on polar bears, students choose their focus, whether it be polar bears or other animals, changing sea levels, temperature and climate, or social changes. The diversity of learners in the class is addressed in two ways--through flexibility of choice in the subject matter so that students can express their interests and identity; and through the differences in reviewer perspectives in the feedback process.

Digital environments also enable students to represent their knowledge multimodally. In the Creator space, students can create multimodal works, inserting audio, video, image and other data types, for example, a 3D animation or a mathematical formula. Such knowledge artifacts, constructed by the learner, go beyond replicating specific content knowledge. Students become active knowledge makers, problem solvers and critical thinkers. Once students submit their revisions, the teacher may request further revisions before publishing the work to an e-portfolio in the student's profile or to the class online community.

This process of crowdsourcing feedback provides frequent opportunities for students to use the embedded rubric in order to develop their understanding of what makes quality writing. In gaining feedback from other students on two to three works, they see other models of how to respond to the question/prompt, which helps them to improve their own works further. They use their collaborative intelligence to develop deep understandings of subject matter and skills to think critically to improve their writing and creative skills.

With explicit criteria, peer feedback is as effective as the feedback provided by the teacher. A study of two Grade 8 classes in a New York City school found that, when students are provided with explicit review criteria, two reviews are equal to a review of an expert assessor/teacher. In a writing project on global warming, where students used a rubric to draft, give and receive feedback, and then revise, there was a low level of disagreement on rating of assessment criteria between peers and even less between peers and an expert assessor (Cope, Kalantzis, Abd-El-Khalick, & Bagley, 2013).

Another form of feedback is through annotations where students can make general comments on a word/sentence/paragraph or specific change suggestions on grammar, punctuation, and vocabulary. An example is shown in Figure 6.

Measuring learning

Technology offers new tools to measure learning by recording and analysing varied types of data. Students' contributions to discussion forums can be analysed in a variety of ways, such as frequency and extent of engagement, language level, and visualisations of the interactions. Student artefacts can also be parsed by natural language processing technologies that identify markers of textual cohesion, vocabulary and latent semantics based on word clustering and frequencies. These data have the potential to be immediately available for formative assessment or to inform planning and instruction for longitudinal analyses, institutional accountability and educational research (Cope & Kalantzis, 2016).

In the example that follows from Scholar (see Figure 7), the teacher has an overview of the class data, including the number of versions, average version length and the average percentage edited for individual students. This is important to identify whether students are using feedback to edit and improve their work, while the academic language levels can also be used to challenge students' vocabulary use. Data on the crowdsourced reviews show the average review rating, students' self-reviews and the optional teacher reviews.

As well, students can invest a lot of work in the reviews that they author and the annotations they provide, so data are collected on these as well. The overall score is based on 22 different kinds of data points; green highlighting and red highlighting show one standard deviation above and below the class norm respectively. This is not because the analytics are norm referenced, but so the teacher can see at a glance which students need support. Once the students improve their work, the analytics can be run again. Teachers can also drill into the data further for individual students, by clicking on the student's name in the left column (blacked out here for student confidentiality) and see version development, peer/self/teacher reviews, and annotations.

Every data point is a teachable moment with immediate feedback the student needs to improve, or the teacher to adapt instruction. These data can be used to target support at any time, not just at the end for summative assessment. They can also be used as evidence of improvement in student writing over time. Teachers using the Scholar analytics commented, 'Analytics are allowing us to have insights that we never had, when with one teacher and a bunch of papers, it was just too overwhelming.'

Many new learning technologies replicate existing practices where assignments and communication with students needs to be uploaded, downloaded and filed. This can be quite time consuming and confusing, especially once the number of files increases. Remember the early days of the school 'G' drive and how soon it became cluttered with files hard to locate. An alternative to uploading and downloading files is to store the work and all communications related to it in one place. The teacher can view the work, check exact times and dates when versions are submitted, have dialogue with the student about the work, check reviews, assign new reviewers/contributors, record notes and any actions such as extra support. Nothing needs to be filed and everything is accessed easily--another time-saver for busy teachers.

Technology is often criticised for increasing teacher workloads rather than impacting on pedagogy (Roberts, 2016). While formative assessment is critical to improving student learning outcomes, informal and structured feedback such as that offered through the discussion forums, reviews and the annotations can be a breaking point for teachers because of the time it takes to offer such feedback. Ask any teacher who has given feedback at the draft and revision stages, as well as at the final submission stage of an assessment piece. Further, implementing more personalised approaches to learning has involved more intensive one-on-one teaching with students, impacting on teacher workloads and wellbeing.

Crowdsourcing feedback, then, is one way to address the issue of increasing teacher workloads: it can be recorded and analysed through machine assessments, it frees the teacher to provide the one-on-one teaching to the students who really need it and to challenge and extend more able students, and it improves student learning.

Learner diversity

The focus on collaboration and social learning in learning environments like Scholar also addresses diversity. This may seem to contradict the emphasis on personalised learning where the focus is on individual learning. However, it is through the interactions that differences are made visible and are used as resources in the learning, creating an environment of productive diversity (Kalantzis, Cope, Chan, & Dalley-Trim, 2016, p. 482). For example, in the structured feedback process and in the online discussions, students see different perspectives, and read and respond to them, sharing insights and prompting deeper thinking. A student commented, 'You could see all the different views from all the different people, it's interesting because a lot of people have different views.'

This collaborative participatory environment also prepares students to be engaged citizens in their communities and in society. Not only are they developing their voice and their skills to think independently, critically and creatively, but they are also developing the attitudes and dispositions to interact positively in communities where there are cultural, social, economic and political differences (Lewis-Spector, 2016). Further, environments such as Scholar, with their emphasis on collaboration, communication, problem-solving, creativity, critical thinking and using technology, are developing the 21st century skills that employers are seeking for a range of occupations, many of which do not yet exist (Masters, 2015).


Technology pervades our lives, and yet many argue that education has been slow to adopt new digital tools. Further, many of the new tools are pedagogically not new--e-textbooks replicate printed textbooks while 'flipped' teaching is in effect a digitised lecture. Whether schools do integrate technology effectively depends on many issues such as funding, internet access and technical issues, as well as teacher motivation, workload and professional learning. Perhaps crowdsourcing, in how it further involves students in giving feedback and collaborating, will be a factor in accelerating change and transforming the way teachers teach and the ways learners learn. Certainly there is much to anticipate in the next generations of digital tools in terms of communication channels, opportunities for collaboration and student agency, access to content, resources, and data and analytics to drive quality learning.


Australian Curriculum, Assessment and Reporting Authority. (2016). Australian Curriculum: English. Available from

Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80(2), 139-148. Available from raisingstandardsthroughclassroomassessment.pdf

Cazden, C. (2001). Classroom discourse: The language of teaching and learning. Portsmouth, NH: Heinemann.

Cope, B., & Kalantzis, M. (2013). Scholar (Web learning environment). Available from

Cope, B., & Kalantzis, M. (2016). Big data comes to school: Implications for learning, assessment and research. AERA Open, 2 (2), 1-19.

Cope, B., Kalantzis, M., Abd-El-Khalick, F., & Bagley, E. (2013). Science in writing: Learning scientific argument in principle and practice. e-Learning and Digital Media, 10(4), 420-441.

Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London: Routledge.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77 (1), 81-112. Available from

Kalantzis, M., Cope, B., Chan, E., & Dalley-Trim, L. (2016). Literacies (2nd ed.). Melbourne, Vic.: Cambridge University Press.

Kline, S., Letofsky, K., & Woodard, B. (2013). Democratizing classroom discourse: The challenge for online writing environments. e-Learning and Digital Media, 10(4), 379-395.

Lammers, J.C., Magnifico, A.M., & Curwood, J.S. (2014). Exploring tools, places, and ways of being: Audience matters for developing writers. In K.E. Pytash & R.E. Ferdig (Eds.), Exploring technology for writing and writing instruction (pp. 186-201). Hershey PA: IGI Global. doi: 10.4018/978-1-4666-4341-3.ch011. Available from

Lewis-Spector, J. (2016). Literary practices for developing engaged citizenship. Australian Journal of Language and Literacy, 39(1), 86-95.

Masters, G. (2015, November 23). A 21st century curriculum. Teacher. Available from ACER website: https://

Mulligan, A. (2011). Trash. Oxford: Random House. Associated learning module available from https://

Roberts, J. (2016). Can technology genuinely reduce teacher workload? Available from Advanced Learning website:

Wiliam, D. (2014, April). Formative assessment and contingency in the regulation of learning processes. A paper presented as part of a symposium entitled 'Toward a theory of classroom assessment as the regulation of learning', at the Annual Meeting of the American Educational Research Association, Philadelphia, PA.

Rita van Haren | English and Literacy Consultant

With over 20 years as a classroom teacher, Rita van Haren has also worked as a literacy leader in ACT Education, focusing on curriculum and pedagogy in preschool-year 12. At the University of Illinois (2013-2015), she developed curriculum, led professional learning and supported US teachers in teaching writing and using Scholar. She served on AATE national council for nine years and is currently on the ACTATE executive.

Caption: Figure 1. Lego crowdsources designs (Lego image--Pixabay, Creative Commons CC0 Public Domain Accessed 24 August 2016 from

Caption: Figure 2. A traditional I-R-E classroom and a collaborative environment (Diagrams used with permission from Bill Cope, University of Illinois)

Caption: Figure 3. Peer review

Caption: Figure 4. Self-review

Caption: Figure 5. The peer review process in Scholar (Diagram used with permission from Bill Cope, University of Illinois)

Caption: Figure 6. Feedback through annotations

Caption: Figure 7. Analytics dashboard
COPYRIGHT 2017 Australian Literacy Educators' Association
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:van Haren, Rita
Publication:Literacy Learning: The Middle Years
Article Type:Report
Date:Feb 1, 2017
Previous Article:Editorial.
Next Article:Changing classroom practice through blogs and vlogs.

Terms of use | Privacy policy | Copyright © 2021 Farlex, Inc. | Feedback | For webmasters |