The unfulfilled promise: formative assessment for collaborative learning online

In their article Brindley, Walti and Blaschke (2009) argue that although assessment has been said to be an important factor contributing to collaborative online learning their analysis did not show indicate that assessment made a difference with regard to participation levels. Despite this rather discouraging result, they nevertheless include assessment as one of ten suggested strategies to enhance collaborative learning. Specifically, they write 

“Monitoring and feedback
The study group conferences and chats are monitored closely by instructors who provide respectful and timely feedback on process and direction when necessary to prevent groups from getting stalled or going off course. Instructors also provide feedback on draft versions of the case studies, and they provide time for revisions before presentation of the final project.” (p. 6)

However, the lack of impact of assessment techniques may be more general than they realize. The review by Black and Wiliam (1998a) is frequently cited as documenting the positive effects of formative assessment for student learning. In the introduction to a popularized version of the paper (Black and Wiliam 1998b) they thus state that:

“Firm evidence shows that formative assessment is an essential component of classroom work and that its development can raise standards of achievement, Mr. Black and Mr. Wiliam point out. Indeed, they know of no other way of raising standards for which such a strong prima facie case can be made.” (p. 1)

In their text, Black and Wiliam (1998) are clear in that they “use the general term assessment to refer to all those activities undertaken by teachers – and by their students in assessing themselves – that provide information to be used as feedback to modify teaching and learning activities” (p. 2, emphasis added). Formative assessment is in turn when the evidence is actually used to adapt the teaching and learning (p. 2, emphasis added).

While they are not very specific regarding the actual content of assessment and formative assessment, two aspects are particularly emphasized. The first concerns the importance of taking the self-esteem of pupils into consideration when providing feedback. They thus argue that “feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid comparisons with other pupils” (p. 6). The second relates to self-assessment by pupils. They believe that this is an essential component of formative assessment, and that it should take the form of “recognition of the desired goal, evidence about present position, and some understanding of a way to close the gap between the two” (p. 6).

The conclusions by Black and Wiliam (1998) can be said to be underscored by the meta-analysis conducted by Graham et al. (2015). They thus concluded that feedback to students in grades 1 through 8 “about writing from adults, peers, self, and computers statistically enhanced writing quality, yielding average weighted effect sizes of 0.87, 0.58, 0.62, and 0.38, respectively” (p. 523). These differences in estimated effect sizes should however not be taken as evidence that there necessarily are differences in the effects of the various feedback methods as the number of studies within each category was relatively small. Still, the estimated effect sizes lie conspicuously close to the often cited range of 0.4 to 0.7 provided by Black and Wiliam (1998).

However, the results from analyses such as these, and in particular those of Black and Wiliam (1998), have also drawn scathing critique (Dunn 2009, Ekecrantz 2015, Elliot Bennett 2011, Kingston and Nash 2011). It has thus been noted that “formative assessment can take many different forms, such as student-reflection activities, detailed student feedback, assessment conversations, and curriculum embedded assessment” (Kingston and Nash 2011, p. 29), and that reviews generally do not distinguish between these and other forms. They consequently provide little useful evidence regarding the efficacy of different forms of formative assessment. Moreover, the concept “formative assessment” is not well-defined, and the distinction between assessment and formative assessment made by Black and Wiliam is for instance often overlooked (Dunn and Mulvenon 2009). (Although note that Dunn and Mulvenon 2009 themselves seem to have missed the distinction made by Black and Wiliam 1998.) Finally, the quality of the meta-analyses has been questioned on the grounds that they have used “untraceable, flawed, dated, or unpublished sources” (Elliot Bennett 2011, p. 5). On a similar note, it has been argued that “rarely focused on feedback leading to students learning something of academic relevance” (Ekecrantz 2015, p. 16).

In sum, while there would seem to be substantial prima facie evidence that formative assessment is a crucial component in any form of modern teaching, the actual empirical evidence for this is less than satisfactory. While it may be no reason to discard the concept of formative assessment, as this may risk throwing the baby out with the bathwater, a more critical appraisal of the concept nonetheless would appear to be called for. The four-point scheme for teacher development outlined by Black and Wiliam (1998) included as its fourth and final point a call for more research, something that against the backdrop of the recent critiques would appear more than overdue.

One of the points that would seem to be particularly interesting regards the impact of different forms of feedback. My personal experience would suggest that although I believe that my instructions regarding an upcoming assignment as well as my feedback following the assignment is unmistakable this need not be the case. Students may fail to comprehend the points I am trying to bring across, both before and after an assignment. This would seem to suggest that feedback is something that needs to be developed in collaboration with the students, rather than something the teacher develops on his or her own. This would obviously also include the use of formative feedback as a way to increase student collaboration - not everything labelled formative assessment is guaranteed to further learning in communities.

References

Black, P., and D. Wiliam (1998a). Assessment and Classroom Learning. Assessment in Education, March 1998, pp. 7-74.

Black, P., and D. Wiliam (1998b). Inside the Black Box: Raising Standards Through Classroom Assessment. Phi Delta Kappan, October 1998, pp. 1-13.

Brindley, J., Blaschke, L. M. & Walti, C. (2009). Creating effective collaborative learning groups in an online environment. The International Review of Research in Open and Distance Learning, 10(3).

Dunn, K. E., and S. W. Mulvenon (2009). A Critical Review of Research on Formative Assessment:

The Limited Scientific Evidence of the Impact of Formative Assessment in Education.  Practical Assessment, Research & Evaluation, Vol 14, No 7, pp. 1-11.

Ekecrantz, S. (2015). Feedback and student learning? – A critical review of research. Utbildning & lärande, Vol 9, No 2, pp. 15-34.

Elliot Bennett, R. (2011). Formative assessment: a critical review, Assessment in Education: Principles, Policy & Practice, 18:1, pp. 5-25.

Graham, S., M. Hebert, and K. Harris (2015). Formative assessment and writing: A meta-analysis. Elementary School Journal, vol. 115, no. 4, pp. 523-547.

Kingston, N., and B. Nash (2011). Formative Assessment: A Meta-Analysis and a Call for Research. Educational Measurement: Issues and Practice. Vol. 30, No. 4, pp. 28–37.