|
Within the range of current investigations into the effectiveness of education and student learning, a wide range of measures have been developed to assess the effectiveness of student learning and other educational outcomes. These measures include data from case studies, student interviews, large scale surveys of student approaches to tertiary education, and cognitive studies of learning style and memory which, together with attainment tests, provide a comprehensive picture of the motivational, attitudinal and cognitive approaches taken by students, and the subsequent achievement in terms of performance.
When the use of CAL and multimedia is considered, the focus of studies of student learning seems to be much narrower. Many of the studies of learning from CAL use a test of attainment at the end of the program as being the only measure of student learning. This raises questions about the nature of such testing, or of the actual test itself: is it primarily a test of recall or does it test the ability to reason; is there a variety of question styles; do the students have to apply their learning to previously unencountered situations; and what is the significance of the learning task to the learner? These questions relate to the effectiveness of the learning task in terms of its value to the student and, while they have wider implications than can be addressed by individual studies, they represent qualitative aspects that have educational significance, but are generally ignored in studies of CAL.
The purpose of this paper is to consider a range of qualitative aspects and measures of student learning in current use, and to consider the insights into student learning that these provide. This will be considered in relation to the measures used to gauge student learning from CAL packages. An approach to CAL that uses qualitative measures appears to be lacking from the research literature. One such approach, carried out by the author, is described. In this study a measure of depth of processing, the Structure of Observed Learning Outcomes (SOLO) taxonomy developed by Biggs & Collis (1982) was applied.
To identify approaches to learning, Biggs (1987), Schmeck (1983), and Entwistle & Waterson (1988) have all developed questionnaires aimed at identifying approaches to learning. An example is the Study Process Questionnaire developed by John Biggs (1987). In this questionnaire, students are asked to report on their normal approach to a wide range of aspects of the learning task. From these responses, it is possible to determine whether the student is intrinsically motivated to learn in depth in order to achieve a level of understanding, or is likely to take an approach of rote learning specific facts. The questionnaire also measures the extent to which achievement, in the sense of high grades, motivates the student. The questionnaire measures approach and strategy on each of the Deep, Surface and Achieving dimensions, which can be simplified to a dichotomy between Deep Achieving and Surface Achieving approaches. By using a questionnaire of this kind, insight can be gained into how particular groups of students approach their study, and also into the range of styles that are used by students in accomplishing a learning task.
Questionnaires measure a basic approach, however other measures have identified that this is not necessarily consistent for each student across every task. Laurillard (1984) found that the depth of learning aimed tor and achieved by students, as identified by a procedure of asking the student to teach back the material that had been learned plus interviewing the students about the strategy used, would vary according to the student's perception of the importance of the task. This means that the quality of learning is influenced to a high degree by whether the student sees the task as worthy of deep study. Some students reported that if they thought the assessment would be a test of memory or recall, then an attempt to understand the material in depth was not necessary, and a more superficial approach would suffice. While the SPQ measures student approach, Laurillard used a qualitative measure of both approach and outcome.
The identification of depth of learning as an outcome has been identified by Biggs and Collis (1982), who developed the SOLO taxonomy to measure this. This taxonomy is used to classify responses to an open ended question, in terms of the relationships the learner draws between the concepts learned and whether the learner is able to structure these according to an appropriate conceptual framework. By using a measure such as this, student learning can be classified according to the extent to which the student has created meaningful associations with the newly acquired knowledge, as opposed to the extent the student has memorised or reproduced the information.
Many of these aspects are the subject of ongoing study by academics involved with research into the quality of teaching and learning, as they are of value tor the insights they provide into what it is about learning tasks that motivate students to learn effectively and perform well.
The argument presented by Hannafin & Rieber, and Jonassen, is that CAL programs are potentially superficial in the mental processes that the learner is required to engage in, and that the resultant learning is likely to be superficial. While this claim has intuitive merit in the sense that it is only by the student making an active response that learning is likely to occur or be effective, it may be oversimplified in that it does not take into account the mental processing that the student may engage in that is not required by the program, which may reflect the student's own learning style or motivation. This issue is investigated by examining whether depth of processing is an outcome even if programs do not, as most don't, use some form of generative learning.
The testing procedure tor depth of learning was carried out after the students had used the CAL program, and independently of it. The students were asked a question about the material they had studied which required an open ended and structured response. This question was assessed using the classification in the SOLO taxonomy, which assessed depth of learning in terms of the relationships the student constructed, and the structured nature of the student's response.
Each study provided a different pattern of responses, with one study showing a high level of SOLO responses, indicating an in depth response by a majority of the students, and another study showing a very low level of student responses. In this second study, most learners reproduced elements of the information provided, without attempting to relate these elements in any way. either to other newly acquired concepts in terms of a structure, or to their own experience in a way that would indicate a process of assimilation with existing knowledge and cognitive structures had taken place. Some answers, showing a deeper, more related, response, were observed. however these were in the minority. These evaluations indicate a contradictory response to the issue, except to say that some students did respond in depth, however the studies were not directly comparable due to contextual and motivational factors that are considered below.
All students were asked to complete the SPQ questionnaire at the beginning of each study in order to provide additional information on the effect of the CAL program on learning. As the SPQ provides a measure of deep learning as an approach to a learning task, and the SOLO taxonomy provides a measure of depth of learning as an outcome of the learning task, a degree of correlation between the two scores can be expected. Studies by Biggs (1979) and Watkins & Hattie (1981) have established a correlation between the two measures, although the correlation is not necessarily strong.
The comparison between the SPQ and SOLO scores highlights the difference between the two groups of students evaluated. In Group A, in which a high level of SOLO responses was recorded, the comparison between the SPQ and SOLO scores shows a level of in depth responses that is higher than the extent of the deep learning approach as indicated by the SPQ scores. In Group B. the level is lower, indicating that many students who are inclined to take a deep approach to learning did not respond in depth.
These differences can only have come about through the circumstances under which the learning took place, including the CAL programs used. Neither program used generative learning strategies, however there were other differences between the programs that partly explain the differences in the results. There were, however, differences in the context in which the programs were used that also partly explain the results. In order to identify all factors that may have affected student performance, additional data was obtained from the students by interview and questionnaire.
Interpretative data was gathered from Group B also, this time by questionnaire. While the responses were not uniform, trends are apparent that indicate the following:
It is difficult to assess by how much, however. The group A students were generally satisfied with the information provided by the program, and found it helpful in reaching an in depth understanding, a self report that is supported by the SOLO data. The group B students found it difficult to obtain the information that was needed, and reported that this did not provide a comprehensive explanation of the topic. These factors were clearly an important influence on the lack of depth that was generally evident in the results from this group.
Biggs, J. B. & Collis K. F. (1982). Evaluating the quality of learning: The SOLO taxonomy. Academic Press, New York.
Biggs, J. B. (1987). Student's Approaches to Learning and Studying. ACER, Melbourne.
Craik, F. I. M. & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behaviour, 11, 671-684.
Entwistle, N. & Waterson, S. (1988). Approaches to studying and levels of processing in university students. British Journal of Educational Psychology, 58, 258-265.
Hannafin, M. J. & Rieber, L. P. (1989). Psychological Foundations of Instructional Design for Emerging Computer-Based Technologies: Parts I & II. Educational Technology Research and Development, 37(2).
Jonassen, D. H. (1988). Instructional Designs for Microcomputer Courseware. Lawrence Erlbaum Associates, Hillsdale, NJ.
Laurillard, D. M. (1984). Learning from Problem Solving. In F. Marton, D. Hounsell & N. Entwistle (Eds), The Experience of Learning. Scottish Academic Press, Edinburgh.
Schmeck, R. R. (1983). Learning styles of college students. In Dillon, R., and Schmeck, R. R. (Eds), Individual Difference in Cognition. New York: Academic Press.
Watkins, D. & Hattie, J. (1981). The learning process of Australian university students: Investigations of contextual and personological Factors. British Journal of Educational Psychology, 51, 384-393.
| Author: Iain McAlpine, University of Southern Queensland, Toowoomba. mcalpine@usq.edu.au
Please cite as: McAlpine, I. (1996). A qualitative study of learning from CAL Programs in two tertiary education courses. In J. G. Hedberg, J. Steele and S. McNamara (Eds), Learning Technologies: Prospects and Pathways, 87-91. Selected papers from EdTech'96. Canberra: AJET Publications. http://www.aset.org.au/confs/edtech96/mcalpine2.html |