|
| [ ASCILITE ]
[ 2004 Proceedings Contents ] |
This project developed as a result of some inconclusive data from an investigation of whether a relationship existed between the use of formative assessment opportunities and performance, as measured by final grade. We were expecting to show our colleagues and students that use of formative assessment resources had the potential to improve performance. This first study, done in semester 1 2002, indicated that there was no apparent relationship even though the students reported how useful they found the resources. This led us to ask if there was a transition effect such that students were not yet working in an independent way and making full use of the resources, and/or whether in order to see an effect we needed to persuade non-users of the resources to become users before investigating if use can be correlated with improvement in performance. With the 2002-3 NextEd ASCILITE Research Grant we set out to repeat our project and to look at use and usefulness of resources in both first and second semester, to encourage non-users to become users and to investigate use with performance. Now our story has a different ending.
In an attempt to address some of these issues, a mixture of online and offline learning and assessment resources have been developed in first year biology at the University of Sydney (Peat & Franklin, 2002; 2003). The assessment resources include both formative and summative items, some with the provision of extensive feedback to individual students. In order to provide flexibility of access and opportunity for independent study, many of these resources are available online through a Virtual Learning Environment (VLE - http://fybio.bio.usyd.edu.au/VLE/L1/) (Franklin & Peat, 2001) and are also available on a CD.
An advantage of online resources is that they are available for use by students any time and any place. In this way, students can be provided with the feedback they are now requesting and this may help them in their final assessment tasks. The development of online assessment resources has a number of advantages over offline pen and paper tasks. The former can be easily marked, often automatically and provide instant feedback to large numbers of students. If so designed, online assessments can be taken repeatedly by students in order to assess and improve their performance. In addition, online tests can be taken unsupervised in students' own time. It has been shown that online assessments allow students to tailor their use to their own learning style, Clariana (1997) and that the use of online formative assessment prior to summative tests can reduce student anxiety (Zakrzewski & Bull, 1999). The contribution of formative computer based assessment on improvements in student learning outcomes is documented by Buchanan (2000) who found that undergraduate psychology students who used an online formative assessment package that provided instant feedback performed better in the end of course summative assessment than those who did not use the package.
While online assessment materials may help meet demands for flexibility in time, place and pace, use limited resources effectively and be enjoyable for users, the ultimate purpose of such assessment resources should be to impact positively on student learning outcomes. In previous work conducted during semester 1, 2002 investigating whether in fact such an impact had occurred, a study conducted within first year biology at the University of Sydney found there was no apparent evidence of any impact of online and offline formative and summative assessment opportunities on student learning (Peat & Franklin, 2003).
Intriguingly, at least some of the students who participated in this study perceived the use of online assessment resources as beneficial to their learning despite the fact that there was no differential effect of the use of the resources on their performance (Peat & Franklin, 2003). This is intriguing because the literature shows that students who make use of every learning opportunity approach the final assessment tasks with a greater likelihood of high performance outcomes (e.g. De Vita, 2001; Heffler, 2001). More specifically, studies have indicated that the use of formative tests before summative examinations increase the grade point of students (Buchanan, 2000; Zakrzewski & Bull, 1999). The University of Sydney is providing a variety of formative assessment resources to first year biology students that would appear to contain relevant feedback yet these resources do not appear to be having the desired impact on student learning outcomes for all students.
The present study aimed to re-examine the question of whether using the formative assessment resources provided in first year biology had any impact on user learning outcomes. In particular, the issue of a possible 'transition effect' on student use of resources and/or learning outcomes was of interest. More specifically, it was proposed that previous findings indicating no impact of online and off-line summative assessment opportunities on student learning might have been due to the fact that the sample were in transition to university. Data collected in second, rather than first, semester of the first year of study, was likely to show a higher level of student self direction to use the available resources.
This study investigates whether non-users of formative assessment opportunities could be persuaded to become users and if so persuaded, was there an impact on their learning outcomes. In order to facilitate usage, the value of using the available resources was emphasised during both semesters to all students. The ways in which this emphasis occurred is detailed below, following a description of each of the four resources and student feedback on each one.
When asked how they thought these modules helped their learning in Biology, a small group of students (n=7) interviewed for this study by an independent consultant generally agreed that the modules "showed us the sort of questions to expect". All seven students agreed that the modules helped with revision of topics and provided a way to test understanding in an interactive way "as opposed to passively reading a textbook". The interactivity of the modules was referred to several times: "it's a dynamic way to do things"; "it adds colour and interaction" "it's more fun".
When asked how these interactive features helped their learning one student reported that "they help you sift through the reading" explaining that they provided a guide to what was important in the textbook. Another student commented, "I played with the CD modules straight away when I got them and all through [the semester] and tested myself at the beginning and throughout". A third reported, "I studied for them, then used them to test my knowledge" and indicated that this strategic use of the modules had helped her "a lot".
It should be noted that the group interviewed were not representative of all first year Biology students nor was the group of a sufficient size so that generalisations might be made. However, student responses to the interview questions provided some indication of the ways in which the resources were viewed by students.
When asked how using this resource had helped their learning, several of the seven students interviewed reported that it helped them determine their strengths and weaknesses in Biology - "[you] know what you remember and what to study". One student reported that studying for the prac exam helped to break up the textbook reading for her. She explained that the preparation for the mid course examination gave her a focus - she went through the textbook and "strategically studied bits that were important". The same student suggested that the marked exam provided "automatic study notes" for later use. Another student agreed, reporting that the answers on the web were "the best thing" because they were so thorough and that they were also a study resource. All seven students interviewed agreed that the "prac exam" helped to consolidate and reinforce their learning, give them a taste of what doing an exam felt like and would be like in terms of format and content and helped them determine which areas they needed to know more about.
The seven students interviewed indicated that the quizzes helped their learning in a number of ways including "reinforcing of what's just been done", ensuring "you really knew what had happened in the lab". One student explained, "The labs are so full. It's hard to sift out what's important, what happened, what the point was. You do three activities - sometimes six or seven activities - and you're not sure what the points are. You think to yourself, 'Why are we doing this?'" Other students agreed and as one put it, "[The quizzes] provide the basis of the main points of what you've done in the labs".
Other learning benefits, from the students' point of view were that "the questions are a link between lectures and pracs"; "[The quizzes] are good examples of MCQs you expect will be on the exam" and "they are good preparation for the weekly [assessed] computer quizzes". One student felt that "Because they are marked on the VLE and you get feedback, this means you have an automatic study tool".
The students found these so useful they suggested their use be extended to include additional questions (to those specific to that lab) that call for revision on previous lab(s). It was suggested that hard copies of the answers be placed in the Biology Learning Centre and further that model answers (hard copy and online) be provided as a means for students to check whether they are "on the right track with the way we are answering". As one student explained - "this would be really good feedback".
With the exception of one student, the interviewees felt these were "fun"; "interactive"; "better than going over the text book" and overall, most useful for testing their understanding of definitions and for revision. One student explained their attraction thus: "if something is fun, it increases your motivation and lightens up study". Another agreed, adding, "you get so sick of reading and then you can't remember what you've read and you think, 'Oh dear me….'". One student indicated the crosswords were perceived as a challenge - "a personal challenge to finish" which another student felt each one "forces you to think about what you've learnt".
After a detailed consideration of student interview responses as well as first year biology teaching staff reflections on the summative resources, it was determined that despite the seemingly discouraging responses from the earlier study (Peat & Franklin, 2003), the resources seemed to hold potential for improving student learning outcomes and that this potential should be explored.
Intervention 1
For semester 1, rather than create yet another sheet of paper or page in the lab book highlighting what there was and why it could be useful, the information was given to the students in the context of the course and interweaved into the lab notes and the lab session activities:
Intervention 2
At the end of semester 1, 2003, students were surveyed, using quantitative and qualitative questions, about their use and perceptions of usefulness of resources. A focus group interview with a sample of students was held, as mentioned above, by an independent consultant to gather student views about the usefulness of biology assessment resources. The interview was designed both to determine why some students did not use these items and the perceived usefulness of the items to those who chose to use them. Students were asked how using each of the item(s) helped their learning in biology and how they thought the item could be enhanced to better help their learning. The student responses led to the development of a colourful information flyer (Appendix 1) that could be used to advertise the resources.
At the beginning of semester 2 all students were given the flyer entitled "Want to improve your marks in Biology??" that advertised the various resources available as well as tips for making best use of them. This flyer was handed out in laboratory classes and was posted on all noticeboards and staff office doors. It was intended that the flyer might encourage non-users of the learning resources to become users during second semester.
At the end of second semester the students were re-surveyed as to their use and perceptions of usefulness of the learning resources. What was of special interest was whether any non-users in Semester 1 became users in Semester 2, possibly as a result of the promotion of the resources and whether this affected their performance. One of the research questions asked was, "Did non-users of formative assessment resources who became users improve their exam performance?". The answer to this research question may help to answer the broader question, "Has student learning been improved by the use of online and offline formative assessment opportunities?" posed in Peat & Franklin (2003).
As much of first year biology teaching is to students who are unlikely to continue with a biological education because they are enrolled in other degree programs (eg Nursing, Pharmacy), it was expected that a student's level of interest in Biology might affect whether or not s/he used the assessment resources which in turn might affect performance. The study sought to examine whether or not there was a relationship between interest in the discipline, use of the available resources and performance.
Each survey collected student demographics, including university entry score, prior experience of biology at secondary school and a rating of their current interest in the study of biology. Participation was voluntary. Perceptions of the usefulness of resources were investigated using a four point scale, with students classifying statements according to whether they used a resource, found it not useful, useful or extremely useful. Open ended questions asked students why they had not used a resource (if relevant) and in what way the resource helped them in their learning (if they had used it). These open ended responses were thematically analysed and categorised (Denzin & Lincoln, 1994). The outcomes of this part of the project are not reported here except to indicate that their responses are very similar to those in the previous study (Peat & Franklin, 2003). Student performance was measured by their final examination mark which included multiple choice and short answer questions.
The implementation of the survey complied with the University of Sydney's Ethics Committee Guidelines for research with humans and this enabled the authors to seek permission to correlate performance with usage/non-usage of materials and with perceptions of usefulness.
Survey 2 was handed out to all students in class time in semester 2. The response rate was 62%. The demographics of respondents in semester 1 were similar to that of respondents in semester 2.
| 2002 sample | 2003 sample Semester 1 | 2003 sample Semester 2 | |
| Self Assessment Modules (SAMs) on the VLE or CD | 79% | 82% | 89% |
| Self test quiz available for each practical session | 82% | 86% | 86% |
| Crossword puzzles | 72% | 77% | 74% |
Non-users of SAMs who became users
One question of interest was whether non-users in semester 1 that became users in semester 2 had moved their position within their cohort when compared to other students. University Admissions Index (UAI) and examination results for both semesters were available for 388 students, who did not differ significantly from the average for the cohort in either semester. The second semester examination marks tended to be lower overall than those for first semester, so to make comparisons across semesters more meaningful, marks were first standardised within each semester for all available student data. The conversion to z-scores sets the mean of each semester examination marks to zero, and an individual's mark then represents the number of standard deviations their mark is from the mean, with above average marks represented by positive z-scores and below average marks represented by negative z-scores. The information captured by the z-scores can be interpreted as relative position within the cohort of 897 semester 1 students, and 811 semester 2 students for whom examination data were available.
The group of interest comprised those students who were non-users of SAMs in semester 1 and became users in semester 2. Therefore, the 388 students were classified into two groups: those who changed to users (n=48), and all others (n=340). Students who changed from non-users to users had significantly higher tertiary entrance scores (UAI mean = 92.0 vs 89.4 for all others, t386 = 2.12, p=.04), so to control for possible confounding effects of academic ability UAI was entered as a covariate into the analysis. A 2 x (2) analysis of covariance with repeated measures on examination z-scores compared those students who changed to users to all other students. Mean z-scores were 0.29 in the semester 1 examination and 0.03 in the semester 2 examination for those who changed from non-users to users, and 0.12 and .04 respectively for others. Although both groups showed a slight decrease in relative standing, the change was not statistically significant overall across groups (F1,385 = 1.74, p>.05). Those students who changed to users did not differ significantly from all others in either relative standing (F1,385 = 0.68, p>.05), or change in relative standing (F1,385 = 1.98, p>.05). UAI was a significant predictor of examination marks, as might be expected for first year university students (F1,385 = 216.9, p<.001).
Overall, it seems that changing from non-user to user of SAMs cannot be directly related to corresponding changes in grades. However, the number of students who fell into the group classified as changing in this manner was relatively small.
All student users of SAMs versus non-users
In order to evaluate the potential benefits of SAMs, the performance of students who made use of the resources were compared to the performance of students who did not make use of them. SAMs use was recorded as a dichotomous variable. In the following analyses, hierarchical regression analyses were carried out for each semester's raw examination mark, first controlling for ability by entering UAI, then investigating whether use of SAMs can be shown to increase performance over and above the contribution of ability. For semester 1 the examination was out of 108 marks and for semester 2 the examination was out of 98 marks.
Data were available for 692 students for semester 1, including 548 users and 144 non-users of SAMs. For the semester 1 examination, UAI made a statistically significant contribution to performance, accounting for 27.6% of variability in examination marks, but use of SAMS was essentially unrelated to performance, with no change in the proportion of variability accounted for (F1,689 = 0.09, p = .76).
For the analysis of semester 2 results, data were available for 588 students, including 518 users of SAMs and 70 non-users. In the semester 2 examination, the use of SAMs made a statistically significant contribution to performance (R2 = .27 for UAI, p<.001; increasing to R2 =.28 with SAMs, (DeltaR2 = .01, F1,585 = 6.33, p<.05). In terms of examination performance, the benefit of using SAMs can be understood as a difference in marks: amongst those of the same ability, students using SAMs scored on average 3.96 marks higher in the second semester examination than those who did not use SAMs.
The differences between results from semesters 1 and 2 may be due to a wide range of possible influences. Just by the nature of time and experience, by semester 2 many first year students are more settled into the patterns of university study and life than they were in their first semester. It is likely that with increased familiarity with and understanding of the systems and requirements of university level study, factors in addition to ability (as indicated by UAI) may increasingly come into play in terms of student learning outcomes. It is possible that by semester 2, some students have begun to employ a range of strategies and resources available to help their study and that these are having a positive impact on their learning. These suggestions are, of course, speculative only but further investigation into the possible benefits of using SAMS seems warranted.
Science students versus other students
Science students were the largest group within the cohort of students studying biology (approximately 56% in both semesters). There were students from other domains (e.g. Nursing, Pharmacy) whose participation in biology was compulsory at the junior level, but who may not have had a great deal of interest in the subject matter. Performance in a subject area might be expected to vary not only as a function of ability, but also of interest in the area. It is possible that those students who are more interested in a subject may make more use of available resources, and it may be that the interest drives performance rather than the resources per se. Students rated their interest in Biology in each semester on a scale of 1 (not at all interested) to 5 (very interested).
The performance of Science students is of particular interest, as these are the students for whom the SAMs were designed in the first place, being the students most likely to continue with the subject area. Science students are more homogeneous in ability (as indicated by UAI) than the other students who study Biology, so that factors other than ability may be more relevant for success in examinations. Effects of interest, ability and SAM usage on examination performance were investigated separately for science students and other students. Table 2 shows the results of the hierarchical regression analyses predicting examination performance from UAI (a percentile ranking), interest in biology (scored 1 - 5), and use of SAMs (no/yes), separately for Science and other students.
| Predictor | Science students | Other students | |||||
| Semester 1 | B at entry(a) | DeltaR2 (b) | p | B at entry(a) | DeltaR2 (b) | p | |
| Step 1 | UAI | .80 | .134 | <.01 | 1.00 | .358 | <.01 |
| Step 2 | Interest | 3.79 | .071 | <.01 | 2.38 | .020 | <.01 |
| Step 3 | SAMs use | -1.04 | .001 | n.s. | 1.12 | .001 | n.s. |
| Semester 2 | |||||||
| Step 1 | UAI | .74 | .117 | <.01 | 1.06 | .413 | <.01 |
| Step 2 | Interest | 2.84 | .047 | <.01 | 3.48 | .047 | <.01 |
| Step 3 | SAMs use | 4.60 | .012 | <.05 | -0.56 | .000 | n.s. |
| a. B is the unstandardised regression coefficient, and indicates the predicted change in performance with an increase of 1 unit in the predictor variable, controlling for other variables already entered. b. DeltaR2 is the change in the proportion of variability in performance accounted for. | |||||||
The final model was statistically significant in each case. For Science students in semester 1, R2 = .21, F3,393 = 33.7, semester 2, R2 = .18, F3,346 = 24.7; for other students in semester 1, R2 = .38, F3,292 = 59.24, semester 2, R2 = .46, F3,234 = 66.6.
The relationship between UAI and performance in examinations was somewhat attenuated amongst Science students given the greater homogeneity of UAIs, and ability accounted for 11.7-13.4% of variability in performance in examinations compared to 35.8-41.3% for other students. Interest in Biology was a significant predictor of examination performance for all students in both semesters, with higher levels of performance associated with higher levels of interest. The use of SAMs made an independent contribution to examination performance in the second semester for Science students but not for those of other faculties. For Science students of the same ability and same level of interest in Biology, there was a statistically significant increase in semester 2 examination performance, equivalent to an increase of 4.6 marks, associated with the use of SAMs.
This study also demonstrated that level of interest in biology can predict examination performance. This, too, may be unsurprising to most teaching academics, whatever the field or discipline. However, interest, unlike ability, can be relatively easily increased. Teacher enthusiasm, efforts to continually demonstrate the relevance of Biology to students' experiences, lives and futures and the use of imaginative and illustrative examples of concepts are a few of the myriad strategies that can be used to engage student interest.
The most encouraging outcome of this study is that for science students both their level of interest in the discipline and their use of SAMs can improve performance amongst those of the same ability. Encouraging students to use the available SAMS, with their dynamic, interactive, "fun" instructive learning based format is likely to increase their enjoyment of and interest in the subject. Ultimately, and most importantly, as this study has shown, encouraging use of the assessment resources is likely to improve student learning outcomes as indicated by their performance, whatever their ability or likelihood of continuing their study of biology.
Buchanan, T. (2000). The efficacy of a World-Wide Web mediated formative assessment. Journal of Computer Assisted Learning, 16, 193-200.
Clariana, R. B. (1993). A review of multiple-try feedback in traditional and computer based instruction. Journal of Computer Based Instruction, 20(3), 76-74.
Denzin, N. K., and Lincoln, Y. S. (Eds) (1994). Handbook of qualitative research. Thousand Oaks, CA, USA: Sage Publications.
De Vita, G. (2001). Learning styles, culture and inclusive instruction in the multicultural classroom: A business and management perspective. Innovation in Education and Training International, 38(2), 165-174.
Entwistle, N. J., Hounsell, C. J., Macaulay, C., Situnayake, G. & Tait, H. (1989). The Performance of Electrical Engineers in Scottish Education. Report to the Scottish Education Department, Centre for Research on Learning and Instruction Department of Education, University of Edinburgh.
Franklin, S. & Peat, M. (1998). Strategies to support learning and student progression: The first year biology way. Proceedings of the Third Pacific Rim Conference - First Year in Higher Education: Strategies for Success in Transition Years, Auckland, New Zealand, (Vol II 37/1-10). [verified 26 Oct 2004] http://www.qut.edu.au/talss/fye/papers98/SF-MP98.DOC
Franklin, S. & Peat, M. (2001). Managing change: The use of mixed delivery modes to increase learning opportunities. Australian Journal of Educational Technology, 17(1), 37-49. http://www.ascilite.org.au/ajet/ajet17/franklin.html
Heffler, B. (2001). Individual learning style and the Learning Style Inventory. Educational Studies 27(3), 307-316.
McInnis, C., James, R. & Hartley, R. (2000). Trends in the First Year Experience in Australian Universities. A project funded by the Evaluations and Investigations Programme, Higher Education Division, DETYA. Canberra: Australian Government Printing Service. [11 Sep 2002, verified 26 Oct 2004]. http://www.detya.gov.au/archive/highered/eippubs/eip00_6/fye.pdf
Peat, M. & Franklin, S. (2002). Supporting student learning: The use of computer-based formative assessment modules. British Journal of Educational Technology, 33(5), 517-526.
Peat, M. & Franklin, S. (2003) Has student learning been improved by the use of online and offline formative assessment opportunities? Australian Journal of Educational Technology 19(1), 87-99, http://www.ascilite.org.au/ajet/ajet19/peat.html
Zakrzewski, S. & Bull, J. (1999). The mass implementation and evaluation of computer-based assessments. Assessment and Evaluation in Higher Education, 23(2), 141-152.
| Authors: Mary Peat & Sue Franklin, School of Biological Sciences, University of Sydney, NSW 2006, Australia Marcia Devlin, Higher Education Division, Swinburne University of Technology, Victoria 3122, Australia Margaret Charles, School of Psychology, University of Sydney, NSW 2006, Australia Please cite as: Peat, M., Franklin, S., Devlin, M. & Charles, M. (2004). Revisiting associations between student performance outcomes and formative assessment opportunities: Is there any impact on student learning? In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 760-769). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/peat.html |
© 2004 Mary Peat, Sue Franklin, Marcia Devlin & Margaret Charles
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.